Speech recognition with image signal

ABSTRACT

A method of speech recognition and person identification based thereon, comprising: recording speech from a speech signal using a microphone; illuminating a speaking mouth; recording a degree of light reflected by the mouth from a reflection signal using a sensor; and recording combined parameters of the speech signal and of the reflection signal, and coupling them to letters associated therewith, per predetermined time duration; comparing a combination occurring in speech of parameters of the speech signal and of the reflection signal to the recorded combined parameters of the speech signal and of the reflection signal which are coupled to letters; and deciding on the basis of the comparison to which letter the combination occurring in the speech of parameters of the speech signal and of the reflection signal corresponds, using block-width modulation of the reflection signal.

The present disclosure relates to speech recognition on the basis ofrecording speech from a speech signal using a microphone; illuminating aspeaking mouth; and recording imagery of a speaking mouth. In the past,it was assumed possible to recognise speech from the recorded sound andimage signals. On the basis of the speech recognition it is possible togenerate written text, enter commands into systems such as computers andmobile phones, and so on.

Such a method is for instance known from U.S. Pat. Nos. 3,383,466,3,192,321 and 5,666,400. These known methods and systems based thereonare however inadequate in respect of reliable and accurate speechrecognition and/or person identification. Further, reference is madehere to WO-97/29481 which is acknowledged as the closest prior art,relative to which the present disclosure is distinguished at least byfeature defined in the characterising portion of the appendedindependent main claim.

With the present disclosure the inventors intend to provide an improvedmethod which enables more accurate and more reliable speech recognition.The present method is distinguished for this purpose by the measures of:recording a degree of light reflected by the mouth from a reflectionsignal using a sensor; recording combined parameters of the speechsignal and of the reflection signal, and coupling them to lettersassociated therewith, per predetermined time duration; comparing acombination occurring in speech of parameters of the speech signal andof the reflection signal to the recorded combined parameters of thespeech signal and of the reflection signal which are coupled to letters;and deciding on the basis of the comparison to which letter thecombination occurring in the speech of parameters of the speech signaland of the reflection signal corresponds. The method further comprisesthe step of block width modulation of the reflection signal. Throughfeatures of the appended main claim in combination, a more reliable andaccurate speech recognition and/or person identification are realized.

The method can comprise the step that the parameter of the speech signalis taken from a group comprising within the predetermined time durationat least: volume dispersion of a difference between a highest and alowest value of a volume of the speech signal; and a ratio of sound ofthe speech signal within and outside a noise level (signal-to-noiseratio). Such a step may be advantageous or beneficial, even without thestep of block width modulation of the reflection signal.

The method can comprise the additional or alternative step that theparameter of the reflection signal is taken from a group comprisingwithin the predetermined time duration at least: an average of thereflection signal; and a degree of increase or decrease of thereflection signal. Such a step may be advantageous or beneficial, evenwithout the step of block width modulation of the reflection signal.

In an embodiment wherein the parameter of the reflection signal is takenfrom a group comprising within the predetermined time duration at least:an average of the reflection signal; and a degree of increase ordecrease of the reflection signal, and the reflection signal hasoptionally been block-width modulated, the method can further compriseof determining the average of the reflection signal as an average overthe predetermined time duration of half the block duration of the blockwave. Such a step may be advantageous or beneficial, even without thestep of block width modulation of the reflection signal.

The method can comprise the additional or alternative step ofdetermining the degree of increase or decrease of the reflection signalas a measurement in degrees. Such a step may be advantageous orbeneficial, with or even without the step of block width modulation ofthe reflection signal and/or camera images.

The method can comprise the additional or alternative step that thepredetermined time duration is 1, 2, 3, 4 or 5 milliseconds. Such a stepmay be advantageous or beneficial, even without the step of block widthmodulation of the reflection signal.

The method can comprise the additional or alternative step ofsubdividing the speech signal into portions corresponding to letters,and indicating at least one of: starting and ending times of letters inthe speech signal; and time durations expressed in numbers of times theshortest duration, which is defined by the frequency of for instance88.200 Hz. Such a step may be advantageous or beneficial, even withoutthe step of block width modulation of the reflection signal.

The method can comprise the additional or alternative step ofdetermining maxima and minima of the reflection signal, and normalizingthe reflection signal on the basis of the maxima and minima Such a stepmay be advantageous or beneficial, even without the step of block widthmodulation of the reflection signal.

The method can comprise the additional or alternative step of: forming,on the basis of the decision of which letter the combination occurringin the speech of parameters of the speech signal and of the reflectionsignal corresponds to, a preselection of potential subsequent letters.Such a step may be advantageous or beneficial, even without the step ofblock width modulation of the reflection signal.

The method can comprise the alternative or additional step of:synchronously registering speech and registering a degree of lightreflected by the mouth in the reflection signal. Such a step may beadvantageous or beneficial, even without the step of block widthmodulation of the reflection signal.

The method can comprise the alternative or additional step of: personrecognition based on recognition from the speech signal and thereflection signal of viseme and phoneme combinations. Such a step may beadvantageous or beneficial, even without the step of block widthmodulation of the reflection signal.

After the foregoing discussion of features according to the presentdisclosure in terms according to the appended claims, a more detailedyet non-limitative description of aspects thereof follows below. In thedrawing:

FIGS. 1-5 show graphs of speech and image signals recorded inconjunction;

FIG. 6 shows a part of the graph of FIG. 5 and conversion into text inmore detail;

FIG. 7 shows a screenshot of a computer program with metadata as anembodiment of a method according to this disclosure;

FIG. 8 shows a diagram of the training of a system as implementation ofa method according to the present disclosure;

FIG. 9 shows a schematic view of speech recognition in practice;

FIG. 10 shows a schematic view of possible applications of the presentdisclosure;

FIG. 11 shows a set-up of a system for implementation of a methodaccording to the present disclosure;

FIG. 12 shows a set-up of a system for implementation of a methodaccording to the present disclosure;

FIG. 13 shows an implementation of a method according to the presentdisclosure;

FIG. 14 shows an implementation of a method according to the presentdisclosure for two different people; and

FIG. 15 shows transformation according to the present disclosure of avariable block wave into an image organ and, finally, further into animage violin.

FIGS. 1-5 show graphs of combined image and sound signals. Recorded onone of the stereo sound tracks in addition to the mono speech sound is asignal resembling a block wave which comprises information about theintensity of the light from a lamp directed at the mouth, in this case adirect current lamp, which is reflected by the mouth. Withoutillumination or ambient light the system does not work, when the mouthis illuminated without ambient light the system operates optimally dueto the absence of noise caused by background light, and in furtherembodiments it may be possible to suffice with ambient light alone, i.e.without a lamp directed at the mouth.

During the letter A the mouth is open and the light reflection isminimal, and during the letter M the mouth is closed and the lightreflection is maximal. The other letters and letter combinations of thealphabet lie there between. During the letter N the mouth is openedslightly, which produces less light reflection than the letter M. Theidea is that the mouth position (area of and around the lips) is anindicator of that which is being said, wherein the viseme (mouthposition) is formed slightly before the phoneme (the letter sound).Analysing the mouth movement in combination with that which is beingsaid results in speech image recognition.

During a training process the characteristic of a phoneme and phonemecombinations can be stored in the letter collection so as to laterrecognize corresponding moments from speech.

Recording Quality Recording Quality of the Image

Recording of speech image takes place by means of block-width modulation(BWM) of the light intensity in the audible spectrum. When speech soundis recorded at 88.200 Hz and processed into index numbers of thismeasurement, the smallest unit is 1 millisecond and is referred to as asplit.

The light measurement of reflection of DC light on lips and the part ofthe surrounding skin around the mouth is higher when the mouth is closedthan when the mouth is open, and the reflection is lower. Backgroundlight from alternating current lamps with a mains frequency of 50 or 60Hz is undesirable here. No light preferably comes from behind thespeaker. Daylight variation is further taken into consideration. Randompeaks indeed move the maximum, and changes in sunlight and slow cloudshadows move the minimum. The sensor is not placed directly opposite themouth due to possible formation of condensation and thermal effect dueto the breath.

Recording Quality of the Sound

Signal analysis with Fourier transformation of 21 successive values andsmoothed into 3 values: first, median and final value, is applied to thespeech sound. The sound file is thereby reduced from 88.200 to 12.600kilohertz. Routines for utilizing fast Fourier transformation are knownto the skilled person, but had not yet been tested at the time offormulation of this disclosure. Fourier transformation searches for sinewaves, as opposed to the block waves in the fast Fourier transformationapplied here, which knows only powers of two as value length. In thecase of 512 values the frequency spectrum ranges from 24 to 6300 hertz.The phase is thus also fully known for speech frequencies of 48-3150hertz.

Four Variables as Building Blocks

A number of variables can be calculated from measurement data as shownschematically in graphs of FIGS. 1-5:

1. Image recording: half the block duration of the block wave of thesensor is averaged over a millisecond, designated in Violin5 with theletter “b”;2. Image phase: in degrees, 0 is the maximum, 90 a decrease, 180 theminimum and 270 an increase in steps of 5 degrees, designated in Violin5with the letters “bf”.3. Volume dispersion: the difference between the highest and lowestvalue of sound in the millisecond, designated in Violin5 with the letter“v”;4. Percentage: percentual ratio of sound within and outside a noiselevel (signal-to-noise ratio), designated in Violin5 with the letter“p”.

FIGS. 1-5 show characteristics of a sound or speech signal 1 recordedwith a microphone and an image signal 2 recorded with a camera. In thesefigures the speech signal and the image signal are synchronized so thatthe present mouth position can indicate an imminent determined sound orphoneme.

The image signal is associated with a degree of reflection of light byand around the mouth, and thus with a mouth position. A level 10 lightreflection can indicate a maximum or a minimum. The degree to which themouth opens less or more is associated with articulated sounds, whichcan thereby be traced back to letters. A transition 5 in the mouthposition in FIGS. 1 and 2 can for instance be detected on the basis of atangent line 11 or a bend line against the path of the graph of imagesignal 2, where a transition between mouth positions can be opted forwhen tangent line 11 is an intersecting line or bend line. The tangentline is a measure of the increase or decrease characteristic of imagesignal 2.

A weak sound signal 1 at or just above noise floor 9, which isindicative of background noise, indicates silence and/or spaces 3, and astrong sound signal is indicative of words 4 being spoken.

In the time between a start 7 of a recording and an end 8 thereof therepresentation of speech signals 1 and image signals 2 is divided intomutually equal time periods, and each time period is designated as asplit 6. These splits 6 each comprise a period of for instance 1millisecond, or a (small) multiple thereof, and are successivelynumbered in FIGS. 1-5.

Speech recognition can be limited to periods or fields of recognitionwith speech 4, between splits four and thirteen and between splitsfifteen and twenty-nine, while silences and/or spaces 3 can be ignored.FIGS. 4 and 6 show how a period from the sixteenth split 6 up to andincluding the twenty-eighth split 6 is highlighted. Data of said fourvariables, which result in metadata after the training phase/karaoke,are recorded for each millisecond or split 6 and for each phoneme orsound.

The data of the increase/decrease characteristic in the image signal andthe average, minimum and maximum values of the speech signal arerecorded for each split 6.

The starting and ending times are recorded for each sound or phoneme,which can take up one or more than one split 6. In a training phase theaverage durations of phonemes or sounds can be recorded, or these can beadjusted in practice.

An average value and a maximum and a minimum of the image signal 2 arefurther determined for each phoneme, as well as an average value of theorientation of tangent line 11, which is also referred to here as theimage phase and/or the increase/decrease characteristic in the imagesignal. This can be done with any desired accuracy, and it has beenfound that a precision rounded off to multiples of 5 degrees can sufficefor very reliable and accurate speech recognition.

The average, maximum and minimum of the volume is recorded from speechsignal 1 for each phoneme or sound, and further also the percentage,which represents a percentual ratio of sound outside and within a noiselevel. 11 metadata are thus recorded for each phoneme, as well asmetadata for transitions between phonemes, which results in a totalnumber of 22. The starting time 12 in the succession of splits 6, when aword begins, is recorded, for instance on the basis of the number of thefirst split 6, in the case of FIGS. 4 and 6 therefore the sixteenthsplit. The same can apply to other points in time, such as beginning andend of phonemes/sounds, and so on.

After a silence or space in which a mouth position stabilizes, astrongly inclining position of tangent line 11 is an indication thatspeech is beginning. Between splits four and five and between splitsfifteen and sixteen a change in mouth position is thus indicative of thestart of a new word.

The image phase is recorded in steps of 5 degrees and providesinformation about the tangent line against the graph of the average halfblock duration, and thus the degree of increase or decrease. In the caseof a closed mouth there is a lot of reflection and the average halfblock duration is short; this is characteristic for for instance theletters b, m and n. In the case of an open mouth there is lessreflection and the average half block duration is long; this ischaracteristic for the letters a and e.

A whole block is a time section 1 and a time section 0. The sensor issymmetrical so that section 1 and section 0 comprise the sameinformation and have (approximately) the same duration, this making halfthe block duration (section 1 or section 0) the smallest informationcarrier.

The image phase is obtained by smoothing the graph of f(t) (averagingover three successive points f′(t)=(f(0)+f(1)+f(2))/3) and marking theminima and maxima and inflection points and having the phase increasegradually (or in steps of 5 degrees) therebetween. This providesinformation from the PWM signal f(t).

When half the block duration is short (in the case of a lot ofreflection), the frequency of the tone of the image violin is high and,when half the block duration is long, the frequency is low. Duringprogramming, it is preferred to work with half the block width.

In order to obtain an equidistant time curve half the block duration isaveraged over a standard time interval of 1 millisecond. Other timeintervals are also possible, for instance 5 milliseconds, or longer orshorter, for which reference is made here to FIG. 15. To be precise, thenumber of half block durations in the time step is counted and the timestep divided by this number is calculated. This then becomes the measurefor the amount of reflected light.

Preparation First Recording Definitions

Viseme is an image component of a word in a language. Phoneme is a soundcomponent of a word in speech. Letter collection: metadata of a letteror phoneme combination with the mouth position associated therewith andthe mouth position preceding it. The metadata are the average of thefour parameters and their standard deviation and their minimum andmaximum.

Determining Predetermined Text

Predetermined text is the selection of words with particularletter/phoneme combinations. The object of the predetermined text is toaccelerate the analysis process, such as image pattern precedingphoneme. Of the 26 times 26 letter combinations, about 30% occurs,approximately 240 combinations. From these, it is possible to choose aminimum number of words, so that the text comprises every existingphoneme combination at least once in order to achieve completeness ofthe letter collection. These words are put in an associated order. Inthis disclosure use is made of the predetermined text “pA mA nA panAmA”(phonetically, short phonemes are written as lowercase letters, anddiphthongs or long phonemes are represented in capital letters).

Writing Out Phonetically

Writing the text out phonetically provides a script for the recording inorder to thereby be able to fill the letter collection, and then be ableto compare subsequent recordings thereto on the basis of such recordingsor to be able to compare them in real time to sound and image recordings(without storage).

Setting Equipment Light Intensity

Determined here are the highest and lowest values of the light intensitywhich is reflected by the mouth by articulating and measuring the letterM (closed mouth gives maximal reflection and minimal block duration) andthe letter A (opened mouth gives minimal reflection and maximal blockduration). This is used for standardizing, whereby the image numberremains within a vertical screen resolution.

Measuring Range

It is checked here, for instance using a mirror, whether the lightsensor records the mouth movements of the lips (the opened mouth mustalso fall within the measuring range).

Sound Volume

The recording sensitivity is here adjusted to the average speech volume,wherein the maximum should only just be able to result in signaldistortion.

Music Sampling

In the recording, sound fields are marked by indicating points in time.The marked field between two points in time consists of a word, space ora phoneme (within a word).

Recognition

There are two forms of recognition:

1. Field recognition of field 14 is shown schematically in FIG. 3 ofpredetermined text (pre-known text), also referred to askaraoke/lettering, from which the metadata can be determined, forinstance during training. Karaoke entails precisely indicating thepoints in time at which phonemes transition into each other. This can bemanual field recognition. The average for the metadata is determinedover these fields, and the most suitable split is then searched for withthe least squares method, which split in turn determines the minimum andmaximum in the metadata.2. Split recognition is also designated for split 15 in FIG. 3, andserves for recognition of spontaneous text, wherein it is not knownbeforehand what the speaker will say.

Average Duration

Since consonants are not are not all the same length and vowels aremoreover variable in terms of duration, a starting and ending time willhave to be given per phoneme, from which an average results for eachphoneme, which is used for a process which is referred to as zipping.This is shown in FIG. 4.

Field Recognition for Karaoke

Moments characteristic for the variable phoneme length are shown asinflection points in the image phase (the minimum, median value ormaximum) and form the index numbers for letter transitions. By removingspaces from the sound, whole words can be ‘zipped’ together. Thestarting times are adapted by zipping. The whole file has a knownduration, so the letter at the halfway point must be correct: bydetermining the starting time of this letter, a part stretches out and apart is compressed in time: this is called zipping with the bisectionmethod.

Determining Metadata

Based on the splits falling within a phoneme field the variables for themetadata are averaged (see table in FIG. 6). A drawback of splitaverages is that there may not be a split which has a determined value.This is why the least squares method is used; the split lying closest tothe average is utilized. These provide minimum and maximum. It ispossible to determine for each split whether the variables fall withinthe extremes of a phoneme or not. This is shown schematically in FIG. 6.

Unique, Multi- and Void Splits

Comparing the splits from the spontaneous text to the metadata of thepredetermined text (particularly the minimum and maximum) results inthree possibilities: the spontaneous split corresponds to only one(unique) phoneme, to multiple (multi-)phonemes, and there are no (zero)hits.

1. Unique splits 13 occur once and can be correct or incorrect. If the Asplit 18 is compared to splits 22, 26 and 27 and the metadata are thesame (because the mouth position A does not change following the priorA), this is a correct split.2. Multi-splits result in a plurality of phonemes falling within theextremes, but are not characteristic enough for unique recognition. Thesplit falls within the min/max range for multiple phonemes.3. Void splits, wherein there is no phoneme for which the split fallsbetween the extremes.

Least Squares

For each split there is a phoneme and two phoneme transitions (one onthe left and one on the right) which is closest on the basis of leastsquares and therefore lies within the smallest (hyper-)cube encompassingthe four variables of the split. In the case that a phoneme transitionis found, two phonemes are being recognized simultaneously (the presentand future phoneme in the case of recognition on the left and thepresent and past phoneme in the case of recognition on the right).

Nearest Neighbour Search

For each of the four above stated variables there is a phoneme which isthe nearest neighbour, which results in four phonemes. If threecorrespond, this is the most likely phoneme.

Optimization of Split Recognition

The correct unique splits 13 are what result in speech recognition. Ifphonemes in a spontaneous text are not recognized, one or morecomponents of the meta-data are incorrect. It is possible to thenoptimize the metadata such that the ‘incorrect’ unique splits ormulti-splits become ‘correct’ unique splits.

This can be done in different ways:

-   -   Move starting and ending times of karaoke and recalculate the        letter collection;    -   Adjust the min/max field for one of the variables in the letter        collection;    -   Have the variable with the greatest degree of distinction be        decisive.

In the view in FIG. 7 of an interface to illustrate the working of amethod according to the present disclosure, “Violin 5” on the line belowthe designations “Split perc vol increase decrease Violin5” shows thephonemes on the basis of karaoke from the predetermined text (imagemouth reflection) for a single person, while in FIG. 14 these are shownfor two people, which shows how reliable the method is, irrespective ofthe person speaking. The phonemes recognized on the basis of splitrecognition are further shown two lines below. The line of graph 16 isof the percentage (p). The line of graph 22 provides the image variable(b) of one phoneme (see also FIG. 13).

Diagram Speech Image ‘Recognition’ First/Predetermined Recording

FIG. 8 shows schematically the method of training of the speechrecognition, with pre-known text, also referred to here as predeterminedtext. As indicated in the description of FIG. 3, the emphasis here is onfield recognition, and FIG. 8 adds that, for training of a system withpredetermined text, the image recognition is emphasized.

Diagram Speech Image ‘Recognition’ Subsequent/Spontaneous Recordings

In the case of spontaneous recognition, i.e. without pre-known text,shown schematically in FIG. 9, the emphasis is on splits as alsoindicated in FIG. 3, and thus the recorded speech or sound signal.

The following key is relates to FIGS. 8 and 9:

The image microphone is the headset with a lamp, (image) sensor andmicrophone directed at the mouth.

The image violin is the block-width modulated signal which is recordedon the second track.

Scopo is a designation for splits with data about the four variables:

-   -   b—image    -   fb—phase image    -   o—percentage    -   v—volume

The designation karaoke stands for the points in time of the phonemetransitions in the articulated predetermined text.

The image organ indicates that when variable b changes, a harmonic pianotone corresponding to the variable b is briefly played.

Letter collection: metadata about the four variables per phoneme andphoneme transition.

The microphone serves for recording speech sound on the first track.

The speech itself is the sound recording in stereo at 88.200 hertz.

The designation spoken text stands for the preceding speech recognitionwith sound on the first track.

Spoken text+image is a designation for new speech recognition with soundand image on both tracks.

Applications Speech-Image Recognition

FIG. 10 gives an indication of various applications of the methodaccording to the present disclosure, which can be inferred from diversepoints in the diagram of FIGS. 8 and 9.

Such applications comprise:

-   -   Tape recorder for recording on two tracks;    -   Speech recognition on the basis of word prediction;    -   Zoom sensor in film or video;    -   Tactile speaker: speech movement for the visually impaired;    -   Mouth reflection: speech visualization for the hearing-impaired;    -   Image organ: speech therapy training program;    -   Speech recognition in noisy environment;    -   Image violin: recognition of all spoken languages; and    -   Letter collection: person recognition for security purposes.

Further information about several such applications follows below.

Zoom Sensor in Film or Video

Films or video with speakers can be dubbed correctly in respect ofvisemes by zooming in on the mouth of the face of a speaker in the filmor video by means of a camera 17 (FIG. 11) with built-in image sensor.This can also be used for recognition of predetermined text orspontaneous text, together with headphones 18 with microphone 19, whichare also shown in FIG. 11. The mutual connections are of lesserimportance to the present disclosure, although a system according tothis disclosure can comprise A/D converters for converting speech signal1 and image signal 2 into digital or at least sampled form, or a systemcan take an analog form.

Tactile Speaker: Speech Movement for the Visually Impaired

The visually impaired can be aided by teaching them to use a tactilespeaker 20. This is a speaker which allows movements or variations inimage signal 2 to be felt at the end of an index finger on a cone 21 ofa loudspeaker, as depicted in FIG. 12.

Mouth Reflection: Speech Visualization for the Hearing-Impaired

The hearing-impaired can be shown image sound, for instance with theprogram Violin5—see FIGS. 7 and 14. The phases of the image variable canbe shown in different colours, for instance white, blue, red and green,depending also on the background colour, which represent the mouthreflection 22, as shown in FIG. 13.

Image Organ: Speech Therapy Training Program

Another application of the present disclosure is a training effect forspeech therapy: stimulating so-called over-articulation. The sound ofthe block wave is inharmonious; this can be corrected by integrating itinto a triangle, which produces a more pleasant sound. This sound, orthe sound of an image organ (conversion from sawtooth to trianglethrough counting, see FIG. 15), provides feedback to the client of thespeech therapist. An image organ converts the image signal into tunedorgan tones; eight per octave, namely C octave+Bes. Such image organswith the option of integrating the block wave into a triangular form areknown to the skilled person.

Image Violin: Recognition of all Spoken Languages

Speech recognition in a noisy environment can be realized for determinedprofessions.

According to the present disclosure, speech recognition can be realizedfor all languages by supplementing the prior speech recognition of onlythe sound with reflection information from imagery.

Letter Collection: Person Recognition for Security Purposes

The speech with image, or at least with light reflection information, tobe recorded according to the present disclosure is unique for eachperson, and almost as unique as—if not even more unique than—afingerprint or iris scan. Consequently, highly reliable, uniquerecognition of a person is achieved by having the person, the speechimage recognition of whom is known, read out a randomly generatedsentence. A system based on the method is then trained. Personrecognition is therefore based here on recognition of viseme and/orphoneme combinations instead of or in addition todatabase-word-analysis, and an example of an interface for this purposeis shown in FIG. 14, similar to FIG. 7 but for respectively one or twopersons, with a recording example of “pA_mA_nA_panAmA” and analysed bythe letter collection and Violin5. Consequently it has been madepossible to combine image and speech registrations with personidentification, to result in a possible double authentication and cantherefore also lead to person recognition and even identification, nextto recognition of phoneme and viseme combinations. In particular whenphoneme recognition has been performed once (or more times) beforehandand has been laid down in a letter collection, even arbitrarily spokenor recited or even read out word combinations may lead to identificationof a person.

FIG. 15 shows conversion of the variable block wave 23, which is digitalat a frequency of 88.200 Hz and can be divided into different durations24. This is converted into the image organ 25. By way of example, 30steps, i.e. 30/88.200, b0-b 1+b2-b3, and so on, are applied here. Theimage violin 26, based on harmonic functions, is obtained by furtherconversion.

Following the foregoing disclosure of aspects, elements, features andfunctionalities of methods of speech recognition are claims defining thescope of protection for this disclosure, comprising at least withindetermined jurisdictions even obvious alternatives to aspects defined inthe claims, which will make it clear that the scope of protection is inno way limited to a specific embodiment or application.

1. A method of speech recognition, comprising: recording speech from aspeech signal using a microphone; illuminating a speaking mouth;recording a degree of light reflected by the mouth from a reflectionsignal using a sensor; and recording combined parameters of the speechsignal and of the reflection signal, and coupling them to lettersassociated therewith, per predetermined time duration; comparing acombination occurring in speech of parameters of the speech signal andof the reflection signal to the recorded combined parameters of thespeech signal and of the reflection signal which are coupled to letters;and deciding on the basis of the comparison to which letter thecombination occurring in the speech of parameters of the speech signaland of the reflection signal corresponds, using block-width modulationof the reflection signal.
 2. The method of in claim 1, wherein theparameter of the speech signal is taken from a group comprising withinthe predetermined time duration at least: volume dispersion of adifference between a highest and a lowest value of a volume of thespeech signal; and a ratio of sound of the speech signal within andoutside a noise level (signal-to-noise ratio).
 3. The method of claim 1,wherein the parameter of the reflection signal is taken from a groupcomprising within the predetermined time duration at least: an averageof the reflection signal; and a degree of increase or decrease of thereflection signal.
 4. The method of claim 3, further comprisingdetermining the average of the reflection signal as an average over thepredetermined time duration of half the block duration of the blockwave.
 5. The method of claim 3, further comprising of determining thedegree of increase or decrease of the reflection signal as a measurementin degrees.
 6. The method of claim 1, wherein the predetermined timeduration is 1, 2, 3, 4 or 5 milliseconds.
 7. The method of claim 1,further comprising subdividing the speech signal into portionscorresponding to letters, and indicating at least one of: starting andending times of letters in the speech signal; and time durationsexpressed in numbers of times the predetermined time duration.
 8. Themethod of claim 1, further comprising determining maxima and minima ofthe reflection signal, and normalizing the reflection signal on thebasis of the maxima and minima.
 9. The method of claim 1, furthercomprising: forming, on the basis of the decision of which letter thecombination occurring in the speech of parameters of the speech signaland of the reflection signal corresponds to, a preselection of potentialsubsequent letters.
 10. The method of claim 1, further comprising:synchronously registering speech and registering a degree of lightreflected by the mouth in the reflection signal.
 11. The method of claim1, further comprising: person recognition based on recognition from thespeech signal and the reflection signal of viseme and phonemecombinations.